Assessing Capability using Mentor® as a Tool

Speaker A Introduce myself briefly since this is my first time at this conference and meeting most of you. I'm with the RBL group. It's a consulting firm founded by Dave Ulrich and Norm Smallwood. We're based out of Utah and we have clients all over the world. And we specialize in leadership development, strategic HR, and capability and organization. We don't use an Ro approach and so I know, but we have kind of, through some of our assessment tools have kind of become introduced to this. So we've been working with Owen Jacobs on a psychometric battery and some applications that we do with that for the last five or six years. And so he's been kind of indoctrinating us into the principles and then we've been kind of coming into more and more contact with the community. So I'm excited to be here and to meet all of you and to learn a lot more about this. And Michael and I are going to talk today about how we've been using mentor to assess capability. And in particular, we're going to talk about in the context of a particular client. So I'll talk for a little bit and then Michael talk for a little bit and we'll probably interrupt each other, or at least I'll interrupt him. So the presenting question that we often have clients coming to us with whether they're designed requisitely or not is kind of how do we get the right people in the door? How do we find the right people for the role and make sure that they're a good fit? And if we don't, there are some consequences to that. And some of those consequences can be particularly pointed in certain kinds of organizations. And so that's where kind of the case comes in. Working with a requisitely designed organization that Michael's been working with, they are in the midst of kind of some strategic transformation for growth, looking at it's a culture where people don't generally move out. So once they're in, they stay. So they want to really be sure that they're getting the right level of people in since if they come in, they're most likely going to stay for a long time. And so I think that's the key points from that. Anything on that? Yeah, please.

Speaker B So the situation again is the organization is moving from level five to level six and for the last 18 months have been concerned and focused on staffing at the L three level. So we are doing an assessment, but in this case, the assessment is used for purposes of selection. We're doing both a personality quote assessment, but we also are assessing for capability. So I know that that's sometimes controversial. Should you assess for capability from outside consultants? I don't need to get into that holy war because in this case it's for the purposes. They are external hires. And unless you can just cast your gaze at them and just look deeply into their eyes and assess what level of capability they are. You might want to use an external kind of tool. Is that okay? Not too heretical yet. Okay, good. Let's keep going. Nothing. I'm going to save Michelle. She's not on tape, so let's continue on. Michelle is saying she's smarter than all of us and can tell capability by looking into their eyes. Keep going.

Speaker A So the organization had an existing battery of assessments and they were confident. There are different criteria used, there are different formulas used for evaluating that. This is kind of some of the typical ones that we tend to see most often. And they were comfortable here with the first four from their existing assessments. But they felt like there were some things that they were missing that they were looking to get a better read on before they as an input into the selection process. And so that's where RBL and the mentor battery come into the picture. And the thing that we've been doing with them is something we call Candidate Position fit analysis. And it combines two different analyses in making a recommendation about the degree of fit and the likelihood of success of an individual for a specific position. One piece of that is the personality profile. Is this something that's going to be simple, easy for them to do, that they're not going to be working against themselves in doing and that they'll find intrinsic reward in doing. And then the second piece is the cognitive capability projection, which is based on, I'm going to say Stratified Systems theory.

Speaker B Close.

Speaker A Okay. And kind of that's done through a resume analysis process where we look at someone's career history and make a projection about where we believe they are right now and where we believe they have the potential to go. And so with that kind of the idea was that all of these criteria then would be met and that they would have a better input and better information in decision making process as part of the hiring. Quick overview of mentor. Happy to answer as many questions as people have afterwards, but sounds like it's similar to where did Marcos go? Yes, to the horizontal stream on the BIOS piece. It's a psychometric battery. It has seven different instruments, which is the exact same number. It's kind of a combination of different types of assessments that we use to kind of cross validate and get a more robust picture. The results are presented through factor analysis. Am I taking too long? Is that why you stood up? Kind of based on some research foundations, adult development, personality, stratified Systems theory, and kind of what makes effective leaders. So those are kind of the foundations of the battery and then the kind of deliverable provides kind of here's the criteria that we think is required for success in the position. Here's how well they fit on personality, here's how well they fit on capability. And here's our recommendation. If there are multiple candidates, the strengths and weaknesses of the two are kind of highlighted. And then sometimes there's a direct recommendation. Sometimes it's, if you want to go in this direction, this is a better fit. If you want to go in this direction, this is a better fit. It just depends on kind of the candidates and what we know about what they're looking for in the organization.

Speaker C Those triangles, are they one person or several?

Speaker A In this case it's one person. Yeah, right. Each person has their own kind of path.

Speaker B It's an analysis of the career history. So we're plotting people just based on resume. No interview on the progression curves. So we've got their age and then just by estimate of reading the resume what that job level is.

Speaker A Yeah. And there's some cognitive measures in the assessment battery and so we use those as kind of does this make sense? Does this seem like it is reasonable?

Speaker B Reasonable sense of their progression over their career.

Speaker A Right?

Speaker D Yeah.

Speaker A So the process is really fairly simple. They take the online assessment, the analysis takes place based on that and the resume and some information about the job that the client provides. Often there's a pre call, particularly if it's not an Ro organization, and you need to kind of find out what level you think the job is at. They can't just tell you this is a level three job, this is a level four job, and then there's a report and a debrief that come with it.

Speaker B So this chart is very clear. So we can just move on to the next one. There's no questions. So I want to just go through I'm going to give you the punchline here. So being the keen researcher I am, I'm the internal consultant and we have a vendor providing us this information, making claims about personality and current applied capability, current potential capability and future potential capability. So mentor is doing all that. As I said, I wanted to know, is it working? So I just went to the line managers and I gave them the report and I said, does this seem to fit? So this is just saying the fun shine is between one and five. The score in terms of the personality description was saying, you know what, you guys have got a good snapshot of you're describing to me the person. It makes sense. Any good assessment tool? Myers, Briggs, FIRO, I mean, Disk, they should be give you a nobody should be really surprised when you get the report back and go, oh my God, they say I'm an extrovert and it should be pretty close. Right. The controversial part was around assessment of capability. Bottom line, our correlations, if you want to go to the next one well.

Speaker A Let me just add something quick before you do that. So this is over the course of the last two or three years, I can't remember exactly, we've done 50 of these. We've assessed 50 candidates and this is kind of basically a year plus after the decision, going back and saying, were we right in what we predicted and what we said back to the hiring manager? And there are 20 cases that we were able to collect data from.

Speaker B And basically what we said is based in the correlation between what the managerial prediction was of current potential capability and what mentors was was zero 80. Our correlation at FPC was less than that, around 6.62, and current applied about zero 66. But around particularly, I was impressed by the correlation for concurrent potential capability. Now I just want to make sure before I run off here, anything else about the particulars here?

Speaker A I don't think probably so we did.

Speaker B Around 50 and 50 of these. I randomly selected 20 people to go back to say, does this report make sense? So I just want to talk about this whole capability issue for a second and start off by saying, dr. Jacobs has been kind enough to work with me and become like, an informal mentor. Owen Jacobs was one of the people that worked with Elliot when Jillian was around as well. We're having a conversation now about something that very smart people have spent their lives studying around for 30 plus years. Elliot jack's seminal work. Owen Jacobs. These people were pioneers. So I just want to be careful. I'll say respectful, as we're talking about this, recognizing that a lot of people a lot smarter than me have tried to figure this thing out. So I think we're all earnestly trying to get our hands on this. And I just want to take a moment to talk about what is it that we're trying to really get at? What are we talking about? I think Jerry Cranes the other day gave an example about one computer chip of the same size and type that could run three different computers. So you have a computer chip that could run because it's designed as part of the computer to produce great graphics, another computer chip that is designed to create music and a third to create calculations. Same computer chip. Here's where I give you my good artwork. Stand back. That's a computer chip. Okay. And here's my question. We talk about gray matter and cognitive CIP. Some people call it IPC. And I'm wondering we're describing the chip, and we talk about it in different ways. And Elliot and others have talked about the fact that this capacity is distinguished by the level of abstraction, the levels of logic. We try to describe what's different about the chip. And while I recognize that people have and we all do, that people have different chips, I wonder if we're focusing on maybe too small a thing. We might be missing the fact that these computer chips, just like for different computers, are used by people to generate different mental models. So the punchline here, the point I'm trying to make, is we want to know about the computer chip, but we're very focused on listening to the frames of reference or the mental models that people build. It's the same thing that you're focusing on. So let me give you a perfect example. This is not abstract. I've got three minutes.

Speaker E You have seven minutes.

Speaker C Okay.

Speaker B I can slow down. Disrespect. We send. I live in Texas. You can tell because I talk real slow. There's some people here that in Texas. I have a brother in law, had never been to Europe before. He's about my age. He's not a young guy and a really smart guy. And he went for the first time, in this case, to China. And I've been to China before. I find China fascinating. He came back from China, said, how was China, man? The food's really bad. Okay? Yeah, the air is bad. Yeah, I know that too. What about what else you got? He says, well, they don't know how to run a factory, and he was there to help them run an aluminum factory. So I taught him the processes. What else, Mike? That's about it. Can't wait to be home. He lives in a world when he goes to China, he comes back. He's dealing with a very concrete world with a limited number of moving parts. He wasn't interested much about the culture. He didn't report back to me about the culture, didn't talk about the business, politics. He's a very smart guy. By the way, that computer chip knows how to generate analyzing a factory and how to make factories faster, better and cheaper. This is the important part that we're listening for in the mentor and in the interviews we do. This is a mental model that's appropriate for work levels one through three. The language is concrete. Their sequencing is concrete, and we know that. But I want to suggest that we change our focus to include not only the type of chip, but talk about what kind of world do they live in. By the way, he fries himself when I ask him, well, why don't you look at maybe taking a staff position inside the organization? He goes, oh, yeah. He says, right. That's strategy stuff. That's where the rubber meets the sky. So not really interested in abstract things, really. Like, you know what? I'm an aluminum plant guy. Faster, better, cheaper. Let me get back to Texas and eat barbecue, and that's it. And those people are important. Okay, here's a quiz. Do those people like leadership development programs and 360 feedback and talking about organizational stuff? No, they don't. You know what they like to do? Faster, better and cheaper and make more aluminum. And you know what we should do with them? Let them do it. Okay, we send somebody else out to China, and they come back, and they see the aluminum plant, but they're also picturing stuff that's not related to the aluminum plant. They're concerned, or they pick up on the organizational politics. They could even maybe pick up on the buca, the external environment, so they just see the world in a more robust way. So we want to know what is people's ability to use that computer chip to make to reflect on experience and to make more increasingly complex models so that they can deal with more increasingly complex work? Makes sense. What mentor is picking up on is a couple things. Personality matters. Number one, some people don't like to make complex mental models. They don't really dig it, the people in this room. Sometimes I think we're blind to the fact that we enjoy learning, we enjoy building more complex models and understanding. You want to know about Marcos? You want to know about Tim? We like to connect these things. Some people don't like doing that. So in the mentor, we're going to pick up on people's proclivity to build increasingly complex models because there are some psychological markers that will show whether they dig that or not. Makes sense. Just go like this and then say, amen. Kim all right, so as a result of the mentor, we're looking at those psychological factors and markers. We don't have time to talk about all them. So I am saying, and as colleagues, I want us to start to think about when we talk about CIP, are we just talking about gray matter? And if so, what? What does that gray matter do for us? What kind of mental models does it build? Levels one through three, which we call the operational domain, levels four and five strategic, or levels six, and then level six and six and seven, strategic. So think mental models. All right. By the way, we use the mentor as a starting point, and we will then have a conversation with people that's along the lines of our colleagues from BIOS. That would be an inquiry into how they see the world. Because you know what some people really want to learn to see? What is it that they don't see? One other piece. We don't have so little time. There's been a whole bunch of work on adult development that we've not heard too much about. Here the work of Robert Keegan. Susan Cook, reuter commons. These people have spent their professional lives, 30 plus years of research, talking about how people construct the world in terms of adult development. I think Owen has done a great job again of looking at some markers that will help us see how do people construct the world anecdotally. My last comment is, this is all fine and computer chips or whatever. We live in Texas. People are very pragmatic, very practical there. My last research based question that I ask them is, did you like this? Did it work? Did it overall fit? And we have pretty Pragmatic managers there, and we've gotten very good anecdotal results to say, you know what? In 18 out of 20 cases, you were right. You missed two. We like this stuff. It makes sense. We'd like to use it again and again. We use it as a basis now for development and going forward. That's it.

Speaker E Up. Do you just want to sit across here?

Speaker B Yeah.

Speaker E I have a question because I'm curious as to where would this fit if it would be a matter of capability, if it's a matter of line or.

Speaker F If it's something else.

Speaker E So, from the past we've had people at the institution that I work at that are very good about what you're talking about right now. Michael kind of putting all these models together. Very great thinkers, very strategic, very good about thinking futuristically, but they just can't land the plane. They can translate that into actually making it happen. A lot of thinking, a lot of great thinking, but cannot translate into results. Would that be I would assume that it's not the capability factor. Is it something else or it is capability?

Speaker C I don't know.

Speaker B I would go back to Elliot's equation talking about applied capability. It's a function of what? Potential plus kses, plus whether they value the work or not. All right. And they get a temperament. So I would look to see if is it a matter of kses or they just might not value that work. They might not think it's very important. I've certainly been in jobs where I was like, you must have me confused with somebody that cares about this. Right? And it showed.

Speaker F Are they very young?

Speaker E Well, there's multiple cases. The case I can think about would be someone probably 40.

Speaker F Okay. Because it could be a matter of maturity, right, of emotional balance. They have very high capability, but they cannot perform at that level because they can't deal with some variables that are more related to emotional balance and maturity. But I don't know if that's yeah.

Speaker E I know it's a very vague case.

Speaker G They might be good people to send information, like thinking they are too much theoretical. Also, we need to see the job and capability. Whether it's a managerial role or individual, there's lots of information.

Speaker D And often someone like that is great in a certain role. So if their job is to come up with those ideas, and then there's a product development team that evaluates them and implements them, then it's probably a good fit. If their job is to kind of be the controller, then that's a problem. And so that's where the kind of looking at it compared to what's needed in a certain position gives you a more accurate view of how close they are to what's needed for that role.

Speaker C So I have two questions. First one is, what is your end? How many people have you run through this thing?

Speaker B We put 50 people through it. I just took a sample of 20.

Speaker C Okay. And the second question is, when you had personality up, it looked like you had nine compound factors. You have leadership and like six or seven other things. So am I to assume or infer that personality as it's being measured, are aggregate scales?

Speaker B I'm going to have Aaron answer that, but yes. So it's an aggregate. There's five multiple tools, but it's based think about it as a framework based on the big five.

Speaker D Okay, so there's seven tools factor analyzed to create five domains of leadership effectiveness.

Speaker C I'm sorry, does it include the neopi?

Speaker D It does not include the neopi. I'm not familiar with that. So I don't even know what's in there that's similar.

Speaker C That's a five factor model. But it has 700 items.

Speaker D Okay, yeah, there is a big five measure, but it's not that one. So then the five domains of leadership that it measures are kind of originality leadership, social presence, surgency, the kind of surgency piece balance between tasks versus people temperament. So calm, easily overwhelmed, that kind of measure. And then I skipped one. What am I forgetting?

Speaker B There's another instrument that is, we ask some questions and there is a writing they've got to respond written with written comments, and that will tell you a lot about how people write, how they formulate their argument. How they write will tell you a lot about their mental models. That's what we're looking for, mental models.

Speaker C Paul yeah.

Speaker B There's a theme here around kind of.

Speaker G Career history, resume analysis, whatever. The resume could say one thing and a career is doing, but what about accomplishment? There could be people that just haven't accomplished anything, but they've kind of gone through this progression.

Speaker B If I could, I'm going to tell you. So I was a little suspect of the instrument for the same reason. So what I would do is I would cheat. I would interview these same people. And when you interview people, you get a lot more information. So I did an accomplishment analysis. So when somebody comes in the same people here, I get to actually talk to them. What really kind of amazed me was how accurate Owen was on his. So Owen and Aaron would beat these up using just resumes. And I thought, they can't do this, but I'm not sure what he was able to see through. But it worked. One other thing I'd like to say is I've been kind of a heretic on this time span and these progression curves, I thought they were like know, hula hoops or fun little items, but I must say that they worked. So I'm beginning to be a believer now, which is to say, when we found out what their job was and we put them on the progression curves, darn things seem to actually work. So I'm like an empiricist if it works. I don't know why it's working, but it's working.

Speaker E Maybe a question back here.

Speaker C So very interesting here about these three approaches. Do you, when you think about assessment, have a view of a possible level of work the tool won't be able to assess? So for example, do you feel that it's good up to level six, that it won't tell you level seven or eight?

Speaker B Well, we talked to our colleagues. What do you say?

Speaker F Yeah, I didn't understand.

Speaker B Does your assessment only does it run out of gas? Does it fail to if somebody's like at a high level, does it maybe not assess them?

Speaker F Oh, no. We have the criteria developed so you can easily assess the highest levels are the easier to assess easier, yeah. The easier to find out at what level of complexity they.

Speaker B Found.

Speaker G The assessment process can address different levels.

Speaker D Yeah.

Speaker C What's the highest?

Speaker G But my experience, mode six. We have had many mode six people.

Speaker F Maximize after two minutes talking, you are like, wow, you can hear the complexity.

Speaker E Of mental processes.

Speaker B In our world. I know some of my colleagues talk about mode eight and nine. They might be that's not my mental model. So the highest we would go up is the seven. I think Owen would agree with that.

Speaker D Yeah. And we have found a few of those.

Speaker E Tina and then Pablo, I have a statement rather than a question. We use DTA process in our organization. And I guess one of the things for me is not to underestimate the importance of understanding capability in order to actually implement structure. Because while it's easy to do the analysis of level of work and understand what the level of work perhaps should be, it is individuals and their own capability that actually realize that level of work. And if you don't match the level of work to the capability effectively, then potentially whatever the theoretical level of work should have been isn't going to be realized because it's going to be influenced by the capability. Buffalo. And then Fred.

Speaker H So two questions to the panel. The one is related when you are doing assessment of people that is working or in the company, how you handle discrepancies? Maybe when manager appraises one thing and your analysis gives another one. And the other question is how you handle feedback to the people that were assessed, not for external candidates referring to people that is working in a company.

Speaker G Yes, there are some discrepancies. Discrepancies, if I can tell you this.

Speaker C Way.

Speaker G One of the causes of the nine box failure is exactly something like that. We have this potential and the performance and they do the committee of managers, they do some assessment. Next year they do again. And the one who was high potential suddenly appears as a low potential where the potential went low. Interesting. And we go further investigated that lot of personal factors affecting the way people look to capabilities is an issue that you must manage because many people with, let's say, MBTI introverted loses impact in the environment.

Speaker H So there is a confusion.

Speaker G Sometimes we do, let's say 100 settlements. We ask the organization, tell me about your 10% of people that are not thinking the way you think, let's look at the body. There are statistical validity not 100% valid. We have people talking about people, but you have a methodology, you have something that support that. You have a theory, you have experience. So we share some doubts with the client and we need to see if there is some. But we never found that let's say this person is level two and should be at level four, should be completely unacceptable, this kind of thing. Then we need to look also to the manager, how the manager is managing.

Speaker E Go ahead.

Speaker F What we find helpful in those cases that we don't find a lot of discrepancies, but when they are, they are usually because they are looking at applied capability. And as you said so sometimes what we find helpful is to involve a one level apparatus. They know the person and that provides a wider view and can help you have a precise well, I was just.

Speaker D Going to say there's kind of two cases that come to mind, and one is like you said, that I think sometimes managers are good at evaluating current applied and not as good at projecting future, particularly if they are a lower mode than the person that they're evaluating. So it's not something they necessarily can kind of imagine. In one client, at least they did a pilot. They gave us about 20 people that they said were all high potential, and they had them go through it. And turns out of those 20 people, only two were actually high potential for kind of, as we would define it, to be able to kind of continue to grow rapidly across different kind of areas, whereas others probably could continue to grow, but would do so more slowly.

Speaker E Have you defined high potential?

Speaker D Yeah, a combination of kind of in this case, it was defined as the ability to move rapidly across very different types of experiences and continue to be successful. And in another case, they were trying to figure out who they should put through a leadership program and they ran like 100 people through it and they had them from three different businesses. And in two of the three businesses, the results that we gave them back really rang true, and they felt like they were right. In the third business, it didn't at all. And as we started kind of digging in with them, it became clear that this was kind of a legacy business and that the people that were successful in this business were actually not the kind of people that they wanted because they were trying to shift the strategy to turn it into more of a quicker growth organization. And so we did a better job of identifying who would be high potential in the new context than their existing managers did because they were evaluating it against the current and or environment.

Speaker E All right, we've got Andre, and then you'll be the last one out of time after that.

Speaker G Michelle I'm allowed two questions.

Speaker C I was quite intrigued by the statistics that the current potential capability seems to be more accurate than current applied. And how are you getting at that? Headroom that difference between the current applied capability, which I understand comes from the resume analysis, and the current potential capability, how do you find it out? The headroom.

Speaker A I can give my shot.

Speaker D And you can correct me.

Speaker A So I think that what kind of.

Speaker D As we were talking about that I think that what the managers are seeing some things in their assessments that we don't see. So they're seeing maybe a knowledge or a skill gap or some technical gaps that is making the disconnect there.

Speaker B Andre they're not privy remember, when they get the resume, they're not really privy to the full scope of the skills that they've deployed. Remember, resumes can.

Speaker C So which was my second question. What checks do you have in the system? Because I just did a quick Google, some personal and anecdotal ample anecdotal evidence. Between 53 and 80% of people lie on their resume. So how do you deal with that?

Speaker A So I guess what I would say.

Speaker D Is we're not looking for the things people expect us to be looking for. And so they can lie about a lot of things and it won't matter.

Speaker A That's the first response.

Speaker D The second one is we're looking at a pattern. We're not looking at any one individual thing. And while someone can inflate their capabilities to a certain point, they're going to show up flat at some point. Am I making any sense?

Speaker B The other way is what they're looking for is scope and scale and the type of work in the abstract or concrete nature of the work. So that's part of the mental model.

Speaker D And we're looking for a pattern. We aren't looking at any one job in isolation and saying, this is the truth of where you are.

Speaker B And I'm not that smart. That's why I need to talk to them. Okay? So when I get in front of them and I can really listen, because.

Speaker D You'Re right, people do fiber close.