I really don't think you have any chance of finding a vibration problem with CFD of any type.  The model of a viscous fluid will only show smooth motion.  The vibration will be a mechanical effect likely with a finite-element analysis of the structure.  In any case, you may need to model the entire ribbon mixer, because I cannot think a way of segmenting the model to get an adequate representation of the forces.  Simply getting the geometry into a model may be a major effort.  Small gaps and fluid near the blades will require a much finer grid than the bulk of the material.
 
Any mechanical vibration will be a result of some unbalanced forces, so any symmetrical model will show only balanced forces.  I am not sure, but Fluent might have a model for a helical ribbon impeller.  However, the model is most likely for a vertical cylinder, not a horizontal trough.  The horizontal model must include a free surface, which is very difficult and time consuming to model.  The fill level of the mixer will have a significant effect on both the forces and the fluid motion.  Horizontal ribbon mixers are rarely used for viscous fluids.  They are most commonly used for free-flowing or slightly-cohesive powders.
 
As with most complicated mixing problems, a physical model and experimental observations are going to be more efficient and reliable than any CFD model, especially for an unusual system.  Most CFD models are developed in conjunction with measurements of a physical model.  Validation of the CFD model will be necessary.  Careful observation and some systematic testing is more likely to identify the type of problems you describe.  I can guess at some possibilites, but they would only be guesses without more extensive information and probably an on-site observation.

The answers by this expert are based on the best available interpretation of the information provided.  The consequences of the application of this information are the responsibility of the user.  If clarification is needed, please submit a further question.