Hi gurus- looking for ideas on how I can load a CSV to a database table without it being so costly. Or is this not how Make is intended to be used?
Source: zipped csv retrieved from API
Target: Azure SQL database table
Yeah definitely not a great use of make. This can f be done directly with your database tools.
Do you think Azure SQL Database can call an API and unzip a file? Where would the file be stored on the serverless database?
Well azure has all sorts of automation.
The native solutions are almost always better if available. And probably more scalable and cheaper in the long run. They won’t charge you per line of your csv. I’m pretty sure of that.
sure, Microsoft Azure has all sorts of automation and their own integration tools (data factory), but then I would ask what the difference is between Make integration vs data factory integration? One only works with Microsoft products, the other seems much more flexible and platform independent.
Here’s how I solved this issue. Much better and only uses 8 operations to bulk load a CSV file to sql server.
3 Likes
That makes perfect sense to use a stored proc to process everything on the server. I was going to mention that after I responded. You just don’t want Make to process the data files especially if they are going to get large. Excellent solution!
1 Like