Commiting data before update in dataflow
Hi, i would like to know if there is a way to commit data before getting update in
dataflow.
We have use kafka to push records to dataset from file. with kafka we can set up how many records we expect so before pushing to dataset we can confirm the number of records in the file and commit it .
Now we would like to avoid having file and push in kind of constant stream the data to our dataset/dataflow.
Let's say our stream data consist of 100 records.
If only 50 records get consume and 50 records are delayed how we can avoid data to be pushed in dataset until all 100 records are ready?
Comments
-
IMHO, easiest way to accomplish that would be to build out that logic in scripting and then at the end of the script trigger data transfer into Domo.
The JavaCLI would be a handy tool for pushing data to Domo
This is the official documentation here
https://knowledge.domo.com/Administer/Other_Administrative_Tools/Command_Line_Interface_(CLI)_Tool
In this tutorial video, I show off one of the many functionalities built into the CLI, renaming datasets and altering schemas.
https://www.youtube.com/watch?v=9HYx1vn3HRM
The CLI command could then be integrated into your PowerShell, Python, java script.
Alternatively, you can trigger WorkBench jobs via the command line:
https://knowledge.domo.com/Connect/Connecting_to_Data_Using_Workbench_5/03Running_Workbench_5#Running_Workbench_from_the_command_line ... so .. same idea just with WB.
Hope that helps!
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
Categories
- 10.8K All Categories
- 3 Private Company Board
- 1 APAC User Group
- 12 Welcome
- 38 Domo News
- 9.7K Using Domo
- 1.9K Dataflows
- 2.5K Card Building
- 2.2K Ideas Exchange
- 1.2K Connectors
- 343 Workbench
- 258 Domo Best Practices
- 11 Domo Certification
- 464 Domo Developer
- 50 Domo Everywhere
- 104 Apps
- 714 New to Domo
- 85 Dojo
- Domopalooza
- 1.1K 日本支部
- 4 道場-日本支部へようこそ
- 26 お知らせ
- 64 Kowaza
- 299 仲間に相談
- 653 ひらめき共有