1-ый способ:
По теореме Виета:
x1 + x2 = 5
x1*x2 = 2
x1^2 + x2^2 = x1^2 + x2^2 + 2x1*x2 - 2x1*x2 = (x1 + x2)^2 - 2x1*x2 = 5^2 - 2*2 = 25 - 4 = 21.
2 способ:
D = b^2 - 4ac = 25 - 4*2 = 17
x1 = (5 + sqrt(17))/2
x2 = (5 - sqrt(17))/2
x1^2 + x2^2 = ((5 + sqrt17)/2)^2 + ((5 - sqrt17)/2)^2 = (25 + 10sqrt(17) + 17)/4 + (25 - 10sqrt(17) + 17)/4 = (50 + 34)/4 = 84/4 = 21
Как Вам удобнее.