Given:
5 digit number = 538xy
Calculation:
Let us first take 54000 as a 5-digit number
Now,
LCM of (3, 7, 11) = 231
On dividing 54000 by 231
⇒ 54000/231 = 233.766(approx)
Now we will find the multiple which can give us a number starting with 538..,
⇒ 231 × 233 = 53823
So, the number will be 53823
Now,
⇒ x = 2 , y = 3
⇒ (x2 + y2) = 22 + 32 = 13
∴ The correct answer is 13.

Alternate Method
As we know from options (x2 + y2) = 25, 10, 17 or 13
Let check option
(x2 + y2) = 25 = 32 + 42
Let x = 3 and y = 4 or x = 4 and y = 3
Number become 53834 or 53843
Check the number is divisible by 3
So sum of digit = 5 + 3 + 8 + 3 + 4 = 23 (not divisible, now we can say this option not correct)
Let’s check other option
(x2 + y2) = 10 = 12 + 32
Let x = 1 and y = 3 or x = 3 and y = 1
Number become 53813 or 53831
Check the number is divisible by 3
So sum of digit = 5 + 3 + 8 + 1 + 3 = 20 (not divisible, now we can say this option not correct)
Let’s check other option
(x2 + y2) = 17 = 12 + 42
Let x = 1 and y = 4 or x = 1 and y = 4
Number become 53814 or 53841
Check the number is divisible by 3
So sum of digit = 5 + 3 + 8 + 1 + 4 = 21 (divisible)
Check the number is divisible by 11
{(5 + 8 + 4) – (3 + 1)} or {(5 + 8 + 1) – (3 + 4)}
(17 – 4) or (14 – 7)
13 or 7 (not divisible)
Check the number also divisible by 53841
5 – 3 + 8 – 4 + 1
14 – 7
7 (not divisible)
Now we can say fourth option is correct because all three option are wrong.
Such number will be = 53823 and it is divisible by 11, 3, and 7.
So x = 2 and y = 3
∴ x2 + y2 = 4 + 9 = 13
∴ The correct answer is 13.
Note:
Choose option 4 directly and solve in the same way, This was the solution wont be lengthy, We have shown here for better understanding only.