Looking for urgent help with API

Spacerat said:
Noted on the effective use of resources. You don't want to bog down your resource by just moving bytes into storage. My largest file being uploaded is 60MB.. so prob not a huge issue.
Not 100% understanding your "ingress -> egress -> ingress". If the app and the webapi is hosted in azure then the app=>api would likely not count as egress? Or is my assumption wrong?
Click to expand...

It's generally not a problem.
But on a project I was working on, we hosted the application in Digital Ocean's k8s managed service, but we stored the actually files in the Australia/Sydney region. So that was ingress into DO, egress out of DO, and ingress into Sydney AWS
 
Spacerat said:
@Solarion , not sure what you do with the files after upload, but make sure that you separate the upload from any processing. Processing can take long (depending on what you do) and can easily make the upload request time out. I upload the file to Azure file share or blob storage and then drop a message into an Azure queue. This message gets picked up by a Azure queue-triggered function and then does the processing in the background, free from time limits.

This is what my upload looks like on back-end
View attachment 1441399
The request
View attachment 1441403

And the MediaTR handler

View attachment 1441405
Methods:
View attachment 1441407

AzureFileShareFileStore:
View attachment 1441411

Hmm looking at it now, I might need to dispose/using the stream I get from formFile.OpenReadStream

I have some client code to do the upload. Just need to find it
Click to expand...

"ErrorOr" - NICE!!!
 
_kabal_ said:
var fileParameter = new FileParameter { Data = fileContent, Filename = formFile.Filename }
Click to expand...

I can't get mine like this. Really hitting head against wall issues with this.

Screenshot_8.jpg

And if I do it this way, the file content is empty on the api side. Oh fyi I'm used to these types of coding shennanigans lol.

Screenshot_9.jpg

Screenshot_10.jpg
 
Last edited:
Solarion said:
Starting to deduce that nswag clients give more problems than they are worth and may just be a lot of hype. Even reverting to basic byte[] file uploads does not work.
Click to expand...
They are amazing, and work wonderfully.

Why write and maintain code that can be generated, is auto updated, and in turn results in compilation errors if a user of that client is not satisfying the API requirements.


If you are having issues with the API client in this specific scenario, what is stopping you from just injecting your own HTTP client, and manually building a body?
 
Solarion said:
I can't get mine like this. Really hitting head against wall issues with this.

View attachment 1441501

And if I do it this way, the file content is empty on the api side. Oh fyi I'm used to these types of coding shennanigans lol.

View attachment 1441505

View attachment 1441509
Click to expand...

Not sure it actually matters, but what if you use "memoryStream" instead of "fileStream"?

Have you actually inspected the stream in your "UploadBackup" method to see that it really exists? Like just call `System.IO.File.WriteAllBytes`

Step through what the API client is then doing, as there is no magic happening, it is plain old c#
 
_kabal_ said:
Not sure it actually matters, but what if you use "memoryStream" instead of "fileStream"?

Have you actually inspected the stream in your "UploadBackup" method to see that it really exists? Like just call `System.IO.File.WriteAllBytes`

Step through what the API client is then doing, as there is no magic happening, it is plain old c#
Click to expand...

I've had to break the uploading down to the basic byte[] to get it to work. Anything with IFormFile and Streams etc I think involves fighting and arguing with Nswag clients generated and after nearly a whole day I'm done arguing.

Screenshot_1.jpg

Screenshot_2.jpg

Screenshot_3.jpg
 
_kabal_ said:
Not sure it actually matters, but what if you use "memoryStream" instead of "fileStream"?

Have you actually inspected the stream in your "UploadBackup" method to see that it really exists? Like just call `System.IO.File.WriteAllBytes`

Step through what the API client is then doing, as there is no magic happening, it is plain old c#
Click to expand...

Ok soooo...this just worked. OpenReadStream()

Screenshot_4.jpg

Screenshot_5.jpg
 
_kabal_ said:
That makes sense I guess

Once you “read” a stream, you are at the end of the stream, and it no long exists.
So in the previous case, the contents of the stream are now in the memory stream byte array
Click to expand...

I literally just out of desperation sat there, looked at the stream parameter and wrote formfile.OpenReadStream() :laugh:

I was not happy with using byte[] as a parameter for an Api. It had to be right. I mean every single tutorial for file uploads as IFormFile so gawdammit it will be IFormFile and I just keyboard mashing guessed it lol Like a pit bull with a car tyre.

Screenshot_6.jpg
 
Solarion said:
I literally just out of desperation sat there, looked at the stream parameter and wrote formfile.OpenReadStream() :laugh:

I was not happy with using byte[] as a parameter for an Api. It had to be right. I mean every single tutorial for file uploads as IFormFile so gawdammit it will be IFormFile and I just keyboard mashing guessed it lol Like a pit bull with a car tyre.

View attachment 1441695
Click to expand...
For small files memory streams are nice because you bypass the whole writing to physical file and clean up. Haven't touched it in a while but we used to also use a using statement around the thing so as to ensure proper garbage collection.

Whether byte array, String 64 or some such, you still need to handle it.
 
Kosmik said:
For small files memory streams are nice because you bypass the whole writing to physical file and clean up. Haven't touched it in a while but we used to also use a using statement around the thing so as to ensure proper garbage collection.

Whether byte array, String 64 or some such, you still need to handle it.
Click to expand...

The only problem I have now is actually unit testing the file upload. Before I could just send the file as a multipartcontent but since I now wrapped the IFormFile in its own class like UploadFileRequest, I have no idea how to actually send this as HttpContent in an HttpClient, as serializing it to a string or json fails.

It all works beautifully on the client side now which is an mvc app, but my unit tests for the file upload on the API side fail. Not sure how to get around this. Have googled my keyboard stikkend.
 
Solarion said:
The only problem I have now is actually unit testing the file upload. Before I could just send the file as a multipartcontent but since I now wrapped the IFormFile in its own class like UploadFileRequest, I have no idea how to actually send this as HttpContent in an HttpClient, as serializing it to a string or json fails.
Click to expand...
If you are testing , store the sample contents somewhere in the test cases and just use them in the test as a emulation, I assume you are only really testing the code behind the upload vs the actual upload itself. You can create a memory stream from anything so if your one argument is the stream, mock/setup one from fixed content and pass it, test your cases.

Disclaimer, I have'nt written Test cases in a loooooong time, principal should stand though.
 
Kosmik said:
If you are testing , store the sample contents somewhere in the test cases and just use them in the test as a emulation, I assume you are only really testing the code behind the upload vs the actual upload itself. You can create a memory stream from anything so if your one argument is the stream, mock/setup one from fixed content and pass it, test your cases.

Disclaimer, I have'nt written Test cases in a loooooong time, principal should stand though.
Click to expand...

I'll give that a crack thanks Kosmik
 
100% DO NOT unit test controller actions, complete waste of time - In my mind controllers are glue. Microsoft guarantees they work correctly, when running through the pipeline. If you "unit test" them, then no middleware will fire, no transformations/validations will happen, so then what really is the point of unit testing them, as the results are meaningless

If you want to test your controllers, do integration tests, and this is really easy to do in .NET 6 - https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-6.0

Then just use the client that nwsag generated to actually call the endpoint.
Then just create a string ("HERE IS MY TEST FILE CONTENT"), and turn it into a StringStream, and pass that as the parameter to `FileParameter`
 
Last edited:
_kabal_ said:
100% DO NOT unit test controller actions, complete waste of time - In my mind controllers are glue. Microsoft guarantees they work correctly, when running through the pipeline. If you "unit test" them, then no middleware will fire, no transformations/validations will happen, so then what really is the point of unit testing them, as the results are meaningless

If you want to test your controllers, do integration tests, and this is really easy to do in .NET 6 - https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-6.0

Then just use the client that nwsag generated to actually call the endpoint.
Then just create a string ("HERE IS MY TEST FILE CONTENT"), and turn it into a StringStream, and pass that as the parameter to `FileParameter`
Click to expand...
Yes agree...
what is cool though is that MSTest can actually instantiate and run the webapi Program class for each test And it is pretty fast at that. Your test then invokes the endpoint as a client would and thus test the entire roundtrip from request to response incl the pipeline, controller, handler, domain, db.... the whole lot. I also tend to test that my handlers return Conflict, BadRequest, NotFound under the correct conditions.
i have some HttpClient extension methods that take care of the heavy lifting of serialisation/deserialization for my tests, but ideally if you include the client class in your test then that is better.
 
I'll post a few of my integration tests against the controllers after LS. Went with webapplicationfactory etc.
 
Solarion said:
I'll post a few of my integration tests against the controllers after LS. Went with webapplicationfactory etc.
Click to expand...
Pairing this approach with the the nuget packages Respawn, Testcontainers and Snapshooter makes writing integration tests very easy, but with high coverage.

Testcontainers will start a MySQL/MsSql/Postgres/whatever container and give you a connection string that you can configure EF with. No in-memory database or sqlite “test” database. You’ll run your EF migrations to init the DB

Respawn will reset your database to a known state. This lets each test start with an empty database, only migrations applied - deleting the database rows is much quicker than starting a docker container.
The tests still run very fast.

Snapshooter takes a snapshot of your test result, and stores it as the known value. The next time the test runs, if the snapshot doesn’t match, it will store it in a mismatch folder. The developer then either needs to fix the code, or make the mismatch snapshot the new correct snapshot. This is great because then you don’t need to perform multiple assertions against your test result.

Now writing a test is very cheap - prepare a payload (optional), call an endpoint, snapshot the result

Really easy to run this in CI - “dotnet test” - and all CI platforms support docker.
 
_kabal_ said:
Pairing this approach with the the nuget packages Respawn, Testcontainers and Snapshooter makes writing integration tests very easy, but with high coverage.

Testcontainers will start a MySQL/MsSql/Postgres/whatever container and give you a connection string that you can configure EF with. No in-memory database or sqlite “test” database. You’ll run your EF migrations to init the DB

Respawn will reset your database to a known state. This lets each test start with an empty database, only migrations applied - deleting the database rows is much quicker than starting a docker container.
The tests still run very fast.

Snapshooter takes a snapshot of your test result, and stores it as the known value. The next time the test runs, if the snapshot doesn’t match, it will store it in a mismatch folder. The developer then either needs to fix the code, or make the mismatch snapshot the new correct snapshot. This is great because then you don’t need to perform multiple assertions against your test result.

Now writing a test is very cheap - prepare a payload (optional), call an endpoint, snapshot the result

Really easy to run this in CI - “dotnet test” - and all CI platforms support docker.
Click to expand...
Ok will def checkout Snapshooter...
On the subject of database... check out https://www.dolthub.com/ GIT for database
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X