If 0 rectangular tables are used, these seat 8 \times 0 = 0 people. Therefore 36 people remain to be seated at circular tables. 36 is not a multiple of 5, so some tables have empty places.
If 1 rectangular tables are used, these seat 8 \times 1 = 8 people. Therefore 28 people remain to be seated at circular tables. 28 is not a multiple of 5, so some tables have empty places.
If 2 rectangular tables are used, these seat 8 \times 2 = 16 people. Therefore 20 people remain to be seated at circular tables. 20 = 5 \times 4, so 2 rectangular tables and 4 circular tables can be used.
If 3 rectangular tables are used, these seat 8 \times 3 = 24 people. Therefore 12 people remain to be seated at circular tables. 12 is not a multiple of 5, so some tables have empty places.
If 4 rectangular tables are used, these seat 8 \times 4 = 32 people. Therefore 4 people remain to be seated at circular tables. 4 is not a multiple of 5, so some tables have empty places.
If 5 or more rectangular tables are used, these seat at least 8 \times 5 = 40 people, so there will be at least 4 empty places.
Therefore the only possibility is using 2 rectangular tables and 4 circular tables.