When one eat or drink with other, you fellowship with those persons. This is the same reason why friend or lovers would go out on a date to eat or drink together. This is also the same reason why a spouse would be jealous when he/she finds his/her partner eating or drinking with someone else. Therefore, when we are having fellowship with pagans (eat and drink) we provoke the Lord to jealousy.

Therefore, my beloved, flee from idolatry. I speak as to sensible people; judge for yourselves what I say.

The cup of blessing that we bless, is it not a participation in the blood of Christ?
The bread that we break, is it not a participation in the body of Christ?

Because there is one bread, we who are many are one body, for we all partake of the one bread.

Consider the people of Israel: are not those who eat the sacrifices participants in the altar? What do I imply then? That food offered to idols is anything, or that an idol is anything?

No, I imply that what pagans sacrifice they offer to demons and not to God. I do not want you to be participants with demons. You cannot drink the cup of the Lord and the cup of demons. You cannot partake of the table of the Lord and the table of demons.

Shall we provoke the Lord to jealousy? Are we stronger than He?

-- 1 Corinthians 10:14-22 (ESV)

It was never about what we eat and what we drink, but about with whom we are having fellowship for what purpose.

Therefore let no one pass judgment on you in questions of food and drink, or with regard to a festival or a new moon or a Sabbath. These are a shadow of the things to come, but the substance belongs to Christ. Let no one disqualify you, insisting on asceticism and worship of angels, going on in detail about visions, puffed up without reason by his sensuous mind, and not holding fast to the Head, from whom the whole body, nourished and knit together through its joints and ligaments, grows with a growth that is from God. -- Colossians 2:16-19 (ESV)

Fellowship is not limited to fellow believers. John wrote:

We have seen and heard we proclaim also to you, so that you too may have fellowship with us; and indeed our fellowship is with the Father and with His Son Jesus Christ. And we are writing these things so that our joy may be complete.

This is the message we have heard from him and proclaim to you, that God is light, and in him is no darkness at all. If we say we have fellowship with Him while we walk in darkness, we lie and do not practice the truth. But if we walk in the light, as He is in the light, we have fellowship with one another, and the blood of Jesus his Son cleanses us from all sin.

-- 1 John 1:3-7 (ESV)

Paul also instruct that we should be careful to pick with whom we partner:

Do not be unequally yoked with unbelievers.

For what partnership has righteousness with lawlessness?
Or what fellowship has light with darkness?
What accord has Christ with Belial?
Or what portion does a believer share with an unbeliever?
What agreement has the temple of God with idols?

For we are the temple of the living God; as God said,

“I will make my dwelling among them and walk among them,
and I will be their God,
and they shall be my people. (Leviticus 26:12)

Therefore go out from their midst,
and be separate from them, says the Lord,
and touch no unclean thing; (Isaiah 52:11)

then I will welcome you,
and I will be a father to you,
and you shall be sons and daughters to me,
says the Lord Almighty.”

-- 2 Corinthians 6:14-18

When believers fellowship with God, it exposes darkness:

For you were once darkness, but now you are the light in the Lord. Walk as children of light (for the fruit of the Spirit is in all goodness, righteousness, and truth), finding out what is acceptable to the Lord. And have no fellowship with the unfruitful works of darkness, but rather expose them. For it is shameful even to speak of those things which are done by them in secret. But all things that are exposed are made manifest by the light, for whatever makes manifest is light. -- Ephesians 5:8-13

Therefore, it is important that fellowship should be built upon the Gospel:

I thank my God upon every remembrance of you, always in every prayer of mine making request for you all with joy, for your fellowship in the gospel from the first day until now, being confident of this very thing, that He who has begun a good work in you will complete it until the day of Jesus Christ. -- Philippians 1:3-6 (NKJV)