The answer is Yes, every star that we see with our naked eye is in the Milky Way. This is true because the spiral galaxy holds every alien planet that humans ever spotted. All objects are seen in the Milky Way including the moon, some stars, and a few planets are part of this galaxy. In diameter, our galaxy is about 100,000 light-years, it is relatively flat and its thickness is about 10,000 light-years. However, there are three to four galaxies that may be seen with a naked eye. These galaxies are seen depending on the position that you look at them from; Northern Hemisphere or the Southern Hemisphere.
They include the Andromeda galaxy, large and Small Magellanic Clouds, and the Triangulum galaxy. Assuming that we look at the night sky from the Northern Hemisphere, the Milky Way seems to get broader in the southern part, the direction of constellations Scorpius and Sagittarius. In the Southern Hemisphere, the Milky Way center is still in the Sagittarius direction but the constellation is closer to overhead. Hence, you can only see the Andromeda galaxy from the Northern Hemisphere and the Magellanic Clouds from the Southern sky.
In conclusion, unless we count seeing the combined light of the billions of stars, we cannot see any other star outside the Milky Way, and therefore, all stars that we see in the night sky belong to the Milky Way.