Imaam Fudayl Ibn Iyaad [rahimahullaah] said:

 ‘Do not sit with a person of innovation, for I fear that curses will descend upon you’

Shaikh Saaleh Al-Fawzaan [may Allaah preserve him] commented on the above statement [of Imaam Barbahaaree]: saying: [That is] because punishment, anger and deviation descend upon a person of innovation, so it is feared that you will be afflicted with some of what afflicts him, and for that reason Allaah said:

وَإِذَا رَأَيۡتَ ٱلَّذِينَ يَخُوضُونَ فِىٓ ءَايَـٰتِنَا فَأَعۡرِضۡ عَنۡہُمۡ حَتَّىٰ يَخُوضُواْ فِى حَدِيثٍ غَيۡرِهِۦ‌ۚ وَإِمَّا يُنسِيَنَّكَ ٱلشَّيۡطَـٰنُ فَلَا تَقۡعُدۡ بَعۡدَ ٱلذِّڪۡرَىٰ مَعَ ٱلۡقَوۡمِ ٱلظَّـٰلِمِينَ

”And when you (Muhammad) see those who engage in a false conversation about Our Verses (of the Qurýaan) by mocking at them, stay away from them till they turn to another topic. And if shaitaan causes you to forget, then after the remembrance sit not you in the company of those people who are the Zaalimoon (polytheists and wrongdoers)’[6:68]

And Allaah [The Most High] said to the believers:

وَقَدۡ نَزَّلَ عَلَيۡڪُمۡ فِى ٱلۡكِتَـٰبِ أَنۡ إِذَا سَمِعۡتُمۡ ءَايَـٰتِ ٱللَّهِ يُكۡفَرُ بِہَا وَيُسۡتَہۡزَأُ بِہَا فَلَا تَقۡعُدُواْ مَعَهُمۡ حَتَّىٰ يَخُوضُواْ فِى حَدِيثٍ غَيۡرِهِۦۤ‌ۚ إِنَّكُمۡ إِذً۬ا مِّثۡلُهُمۡ‌ۗ

إِنَّ ٱللَّهَ جَامِعُ ٱلۡمُنَـٰفِقِينَ وَٱلۡكَـٰفِرِينَ فِى جَهَنَّمَ جَمِيعًا

‘And it has already been revealed to you in the Book (this Qur’aan) that when you hear the Verses of Allaah being denied and mocked at, then sit not with them, until they engage in a talk other than that; (but if you stayed with them) certainly in that case you would be like them. Surely, Allaah will collect the hypocrites and disbelievers all together in Hell.’ [4:140]

In this is a warning against sitting with the people of misguidance and desires- [a warning] against sitting, accompanying and listening to their speech or reading their books. So it is upon you to be distant from these affairs. And Allaah’s Aid is Sought! At present the one who acts upon this [i.e. the one who keeps away from the people of misguidance and desires] is said to be a secluded person and a boycotter, and that he is one who harbours doubt against the people.

[It’haaful Qaaree Bitta’liqaat Alaa Sharh As-Sunnah Lil Imaam Barbahaaree.page: 318-319 Vol: 2). slightly paraphrased]