Did you know that Microsoft has a brute force mechanism in place for Azure SQL Server? It offers users more protection if someone wants to break into the database. In this article, we take a closer look at the problem and how we arrived at the fix.
Probleem?
When deploying an API application in Kubernetes, incorrect credentials were added to the configuration file. A retry mechanism was not taken into account in code, causing the application in the Pod to keep trying to reconnect to the SQL Server.
The application kept receiving a “Login Failed for user ‘fakeuser'” from the SQL server. Through this Pod an attempt was made to connect to the database via a sqlcmd command, it kept giving the same error message except on the master database.
After creating a new user, the error message remained unchanged and connection could only be made to the master database.
After spinning up a debug Pod, click here for more information, we did manage to connect to the correct database, even from another jump host SQL Management studio managed to connect.
After all my ideas ran out and my colleagues did not have an immediate idea about what the possible solution could be, I wanted to try one last time to find the root cause. I started digging into the Logs of the Azure SQL Server. In these I noticed that there were many attempts from the same user. There was a slightly detailed error message that popped up namely.
error_state = 13
Solution
After I found the error message I started searching Google for more details. And I came across the following article on stackoverflow:
sql server – Getting SQL login failed error (18456) when connecting to Azure SQL – Stack Overflow
Apparently, Microsoft has a mechanism that if you try to log in many times the requester’s IP is blocked for 30 minutes. After we deleted the Pod and deployed a new one with the correct credentials the problem was solved.
Hard lessons learned here! 🙂
