I don't normally use the 40 weight thread in the bottom, usually I use a lighter 50 weight like Aurifil, or even lighter like 60 weight superior so fine in the bobbin. That might be part of what's happening- the heavier cotton thread will pull too much. The reason that the 40 weight polyester thread works better is that it's probably somewhat thinner and also less "linty".
Every brand of machine has differences in what top tension is needed. For my old Pfaff, I needed tension 1 or even 0. The Janome needed tension 8 or 9. They had a special bobbin case set to a lower bobbin tension that you could use for free motion quilting so you could lower the top tension. It wasn't necessary, however, it was easier to remember which bobbin case you were using
. My current Babylock likes tension to be around 4 to 5. You might google Viking free motion quilting to see if other Viking users have some thoughts about tensions.
It's possible to change the bobbin tension, but a little goes a long way there. If you decide to try that, take a picture of your bobbin case before so you remember how to get back to its usual setting. Probably working with a different bobbin thread and playing with the top tension will help so try those first.
Happy quilting,
kathy