Fact Check: The NCARB/Lineup Connection

Comments

33 comments

  • Avatar
    Alexander Gault

    Facts are great and can shed light on otherwise dark subjects. Candidates often feel in the dark.

    Who is your 'Psychometrician experts'??

    Taken from NCARB blog "A 'clean desk' policy is recommended by testing experts. The replacement of scratch paper follows best practices in testing recommended to NCARB by our psychometrician experts. Requiring a 'clean desk' is the most reliable way to ensure that candidates are not violating the ARE Candidate Agreement."

    Taken from Prolydian website "The testing environment must be 'clean' and proctors are trained for this assessment. Additionally, the proctor has the candidate upload a government-issued ID"

    Let's steer the conversation away from leadership income theories which you just de-bunked. (That's not the story that matters and/or will return us to pencils & paper). The internet shows Lineup & Prolydian are now merged. Psychometrics is a 'popular feature' at Prolydian. Please prove your expert recommendation was not provided by your own subsidiary company. Sounds an awful lot like Prolydian IS involved in the delivery of the ARE b/c they took away the pencils.

    Furthermore, the 'clean desk' policy should be vetted by outsiders and tested before roll-out. Perhaps showing AIA board of directors, State Boards, Psychometric companies without financial gain, and the actual Candidate testers themselves.

    Nobody is attacking your leaders; just shed light on the secret recommendations, and if you think that 'we the people' should accept that as gospel.

     

    9
    Comment actions Permalink
  • Avatar
    JZurn71

    Let me clarify a few things regarding the development of the ARE and consultants that are engaged in the development and delivery of the ARE.

    • Test development and psychometric efforts are provided by Alpine Testing Solutions. There organization employes multiple Ph.D. experts in the area of standardized testing and measurement.
    • Candidate management services are a combined effort between NCARB and Alpine Testing Solutions. As a candidate, you log into an NCARB portal that uses Alpine Testing Solution's candidate management application to manage exam eligibility, record final scores, produce score reports, etc.
    • As a candidate, you schedule your appointment with Prometric and Prometric provides the secure test center environment in which you test.
    • When you are at a Prometric center, they launch the exam via the web from a company called Zoomorphix. The Zoomorphix application delivers the exam and scores your results which is what allows a provisional score to be provided in the test center. Once your exam is complete, your results are sent from Zoomporhix to Alpine Testing Solutions which does the final processing and produces the score report.

    It is the Zoomorphix application that controls the calculator and other tools during testing. It will be this same application that supports the whiteboard.

    There is no Prolydian involvement in the ARE. As noted in the original post, the only thing the ARE department uses Lineup for is to coordinate having a diverse group of architect volunteers work on writing and reviewing the ARE.

    It doesn't surprise me that the Prolydian website (a website I've never been to) has similar language as shared by our psychometricians because a clean desk policy clearly is best practice across the testing industry.

    -1
    Comment actions Permalink
  • Avatar
    JZurn71

    Alexander,

    Following up on your note, "The internet shows Lineup & Prolydian are now merged." I can confirm that is not accurate.

    Lineup and Prolydian did integrate their applications but that doesn't mean the companies are merging.

    -1
    Comment actions Permalink
  • Avatar
    JZurn71

    Laura,

    The creation of a for-profit subsidiary is not illegal and, in fact, is an appropriate way to separate the assets of the non-profit NCARB from the for-profit Lineup. If NCARB had not separated the workings of Lineup into its own corporation, NCARB would take on the responsibility of paying unrelated business income tax on surplus revenues generated by Lineup.

    Your statements about non-profits are correct in that no individual can have private gain. Unlike architectural firms, where owners can take any surplus revenue generated as personal gain, in the non-profit world, the surplus revenue generated is funneled back into the development and delivery of other programs.

    The structure of the Lineup subsidiary is beneficial as it is a wholly-owned subsidiary of NCARB. This means any surplus revenue flows directly back to NCARB and can again be directed to support programs and services. It is important to understand the NCARB individuals serving on the Lineup board do not receive compensation from Lineup as a corporate entity - they serve on the Lineup board to ensure the interests of NCARB are taken into account.

    0
    Comment actions Permalink
  • Avatar
    JZurn71

    Leah,

    Alpine Testing Solutions advised NCARB to eliminate the use of paper and pencils in the test center upon the launch of the online proctoring option. Knowing that many candidates do use the paper and pencils, NCARB had to pivot to developing an electronic alternative.

    The consistency of tools across delivery modes is critical to ensure that the exam is fair for all candidates. If we cannot support paper and pencils for online test takers - we don't support paper and pencils for onsite test takers.

     

    -1
    Comment actions Permalink
  • Avatar
    JZurn71

    Leah,

    I hear what you're saying and understand that right now some candidates have lost a level of trust with NCARB. You may have absolutely no trust in NCARB.

    The rollout of the news about scratch paper has not gone well and many conversations have taken place inside NCARB and with the Board of Directors over the past few days. The reality is that several changes have to be made to support the online proctoring option. I think candidates will appreciate some of the changes and I know candidates will not appreciate others.

    All that being said, the development, delivery, and ongoing evolution of the ARE will continue to keep validity, fairness, reliability, and defensibility as the top issues. And, yes, at times to maintain these aspects, decisions will be made that will not be appreciated by candidates.

    -1
    Comment actions Permalink
  • Avatar
    JZurn71

    Leah,

    As noted above, It is the Zoomorphix application that controls the calculator and other tools during testing. It will be this same application that supports the whiteboard. Alpine Testing Solutions provides psychometric and test development services along with aspects of the candidate management system.

    In regards to paper/pencil usage at work vs. in the test, I will clearly acknowledge that the test is not a direct reflection of practice, never has been, and never will be. It is a knowledge test that assesses the critical aspects of practice. The task aspects of licensure are addressed through education and experience.

    There will be consistency in the overall measure of competency even with the switch to an online proctoring option. The changes being implemented for online delivery are resulting in a re-evaluation of the cut score to ensure that the consistency is maintained. It will be a different test experience and we certainly will have to adjust the cut score accordingly.

     

    -1
    Comment actions Permalink
  • Avatar
    JZurn71

    Leah,

    As for the date selected, that was an effort to be responsive to the candidates that are not comfortable or able to go to a test center due to concerns over the pandemic. Many candidates have not tested since the pandemic hit in March. NCARB wanted to provide them a testing option as soon as possible.

    -2
    Comment actions Permalink
  • Avatar
    JZurn71

    Leah,

    I'm sorry if you think I didn't properly address your question:

    How was this Whiteboard developed and was NCARB in the room when it was made?

    • The specifications for the whiteboard were outlined by NCARB staff architects. The development of the whiteboard is being done by Zoomorphix as part of the delivery tool. The application is still being developed and refined.

    The comment about the cut score changing is entirely factual and not a carrot. It is the reality of exam evolution. I cannot tell you exactly how the cut score will change as, of course, the outcome of such a decision has yet to be made. The process by which the cut score will change is called equating. I will not write a ten-page paper on the subject as is possible but will summarize as:

    • the overall level of difficulty of future items as delivered after the launch will be compared to the overall level of difficulty compared to today
    • if it is found that items are performing as more difficult the cut score will be lowered, if it were to be found that items are performing less difficult, the cut score may be raised
    • this is a comprehensive process that allows all changes to the exam to be taken into consideration
    • the evaluation will happen independently for each division as each division functions independently

    I do understand that many candidates would prefer paper and a pencil when taking the ARE. To ensure fairness for all candidates both online and onsite, the ability to take notes and sketch will be available through the whiteboard tool.

    0
    Comment actions Permalink
  • Avatar
    Lilia Grigoryan

    Jared,

    Clearly NCARB isn’t sympathetic to Leah or other candidates.


    Before talking about fairness, maybe you should start from your organization being more transparent to the exam and the candidates. Simple example, why doesn’t your exam indicate what percent of the test was incorrect to receive that score instead of an arbitrary pass/fail w/ no explanation. How do we know it’s not a system error or glitch? There are many people who left this profession because of your organization. Just take a look throughout this forum... you think you were fair to them by letting them down.

    You know people afraid to express their opinion in this forum because they think you will fail them again. Don’t you you see there is so little traffic in this forum before this whiteboard controversy.
    The candidates simply don’t trust NCARB.

    You can fail me again, I am already numb.

     

     

    9
    Comment actions Permalink
  • Avatar
    Heather Woodcock

    Jared,

    I'd first like to thank you for being a human in this distressful time and allowing us to have a discussion, whether it proves positive or negative. You're answering the questions the best you can, and I am glad NCARB has realized how drastic this change is to us and is (hopefully) taking appropriate action.

    Please take this to your NCARB team:


    WE the test takers would like a digital format that matches that of pen and paper. If it means we have to strap a Wacom (or similar mouse trackpad like) tablet in so we can easily control the mouse on the screen like its the tip of a pencil, so be it. I think that would be the best case scenario to make literally everyone happy. WE the TEST TAKERS would like to be able to see this whiteboard on the screen 24/7. It cannot vanish, cannot be automatically erased, cannot glitch, and cannot cause us stress. Paper does not cause us stress. We will need tools within this whiteboard that accurately allow us to do complex structural calculations and draw crude diagrams within seconds. Every second counts on these exams and it's important that this whiteboard works with us, not against us.

    8
    Comment actions Permalink
  • Avatar
    JZurn71

    Lilia,

    The reason the ARE does not use percentages for descriptive feedback is that the percentage of items needed to pass varies from division to division and can vary based on the set of items you receive on test day. This is why NCARB provides descriptive feedback using the table provided on page two of the score report. A candidate's level of performance as reflected on that is a more accurate reflection of areas of strength and weakness.

    When a candidate passes, no descriptive feedback is provided as the candidate has been determined to be proficient in this division.

    I'm sorry if my answers are being perceived as analytical and not acknowledging the emotional impact that I do know the exam process has on candidates. It takes time, it costs money, it is stressful, it can certainly seem ambiguous, it pulls you away from family and friends - I really do get it. I am trying to provide honest information about exam development and delivery. If you want to talk through the best way to understand the descriptive feedback, I can certainly do that with you.

    0
    Comment actions Permalink
  • Avatar
    JZurn71

    Heather,

    Will do! I've already copied your note and will hit send on the email next.

    1
    Comment actions Permalink
  • Avatar
    JZurn71

    James,

    As I'm sure you can understand, NCARB is trying to balance providing access to the ARE for candidates that currently are not comfortable testing onsite or have underlying health conditions that limit their ability to test with releasing an online proctoring option.

    I am reading, listening, and relaying.

    0
    Comment actions Permalink
  • Avatar
    elifbayram

    Jared,
    I think the biggest question here is: How can NCARB promise that the whiteboard feature will not crush the exam? Will not cause delays or lagging..? Since you said the company who is delivering the exam will deliver Whiteboard feature too, I am beyond concerned for test takers. As a person who recently passed all of the exams, there was not one single exam that I took that didn’t have a software issue during my exam. Me and many other people I know could not use the Highlight or Strike tools due to fear that it will crash the software which is a fact. Are you aware that people who passed exams recommend everyone who is testing to not to use this features..? How sad is this!?! Why do we have those features if we are not gonna be able to use them? What makes you think that Whiteboard will be exempt of this issue.? In the year of 2020, how/why are we still talking about this archaic technical issues? And more importantly why NCARB haven’t taken any steps to solve these technical issues so far? Again , at this age of technology, exam crushing, computer reboots, online tools being unavailable and so on are beyond unacceptable. Can’t NCARB find a company who can deliver a better technology..? As a person who have recently created her own online testing website and delivering online quizzes, I became familiar that it is expensive but possible. Even the generic software that I am using does a quite okay job. I am sure NCARB can (should) invest in a better technology.
    So going back to where I started, I think even if people can wrap their had around this new “paperless” reality (which clearly they hate the idea) what makes NCARB think that same tech company will be able to provide this technology sufficiently.?

    Elif Bayram
    info@arequestions.com

    8
    Comment actions Permalink
  • Avatar
    Richard Balkins

    Jared NCARB and for everyone else reading.

    The Zoomorphix app that Jared is referring to is called Exam Studio

    Click the link to learn more about Exam Studio.

     

     

    0
    Comment actions Permalink
  • Avatar
    Richard Balkins

    Jared is correct in this: clean desk policy

    Even the proctors and psychometrician consultants that AIBD used still recommends a "clean desk policy" within the context of authorized and non-authorized. While the AIBD's exam is open-book but since it is a 3-hour exam that involves a lot of subject matter, which is fair to make the exam passable. We have the many subject areas because it also allows us the flexibility to divide the exam into "multiple" tests similar to NCARB but because of the cost of administering like 4 exams or something, it was cost-prohibitive but you still are tested in a rigorous exam where you can be asked questions of all kinds of subject matter. It isn't the length or number of questions you are asked but the breadth of subject areas from which you can be asked questions and large pool of test items that can be asked in each area and you really have to be able to jump between subject areas in the 3 hours. 

    Even then, a desk should have as little clutter at all if any and only what should be allowed. Since we are a closed-book exam with the ARE, we should have even fewer things at the desk. This is fine and to be expected. This is expected in ALL 'closed book exams" because you would not have text books. 

    Removing scratch paper is partly meant to keep your eyes on the screen because any camera (whether remote proctoring or at a test center) is looking at you and can see your head from forehead to chin, at ALL times. This move to digital 'scratch paper' from physical scratch paper has nothing to do with ARE but exam proctoring in general. Proctoring happens everywhere from SATs and similar testing at grade schools including classroom exams that these companies also provides exam proctoring services for school districts through college/universities, through to licensure / professional certification exams. It has been found that people have been cheating everywhere in world of exam proctoring and some of that cheating involves physical scratch paper. When I wrote a sort of allegory on another thread on the ARE blogs/etc. "little Johnny" cheating on his "math exam" could be responsible for the changes to digital whiteboard, that allegory isn't too far from the truth even if indirectly. 

    Proctoring policies of proctors are based on a larger holistic evaluation of proctoring exams not just ours but ALL of them and what they find people cheating and how they cheat and policies to combat that because if they would do it for a medical exam to become a doctor, why would people taking the ARE be any different. Sure, not everyone taking exams will cheat but it always is a relative few but they do great harm to the exam and they often do great harm. There are cheaters in every exam especially exams where it is compulsory and required in order to attain both the professional title and to independently practice. 

    NCARB still has authority over what they permit 

     

    -1
    Comment actions Permalink
  • Avatar
    Leah Underwood

    Richard? Have you taken these exams? If so, how did you do?

    1
    Comment actions Permalink
  • Avatar
    Richard Balkins

    Just to be clear, there are multiple parties.

    1. Alpine Testing Solutions is a psychometric/proctoring & test development consultant company. They are not a software developer but they are consultants involved with NCARB.

    2. Together, they (NCARB and Alpine) developed specifications from which the software development company(-ies) may be tasked to meet with their software. For example, because of recommendations to switch to digital whiteboard was made by Alpine Testing Solutions and ultimately policy changes were approved by NCARB, NCARB then directed there exam software development consultant Zoomorphix Systems to add digital whiteboard feature.

    3. A number of specifications were made regarding features to be on the digital whiteboard by staff architects at NCARB which is directed to Zoomorphix to implement.

    NCARB, am I on-track with my understanding?

     

    1
    Comment actions Permalink
  • Avatar
    Leah Underwood

    Aw...
    You don't have to tell me how you did if you're uncomfortable, just which ones you've taken.

    0
    Comment actions Permalink
  • Avatar
    Richard Balkins (Edited )

    Leah,

    I'm just catching up on the posts.

    I am an ARE exam candidate (via state of Washington licensing board) waiting for the online proctoring for ARE since proctor sites are kind of a significant distance from my location but I recognize the concerns given by people.

    I can take the CPBD exam at any time but my familiarity with the CPBD exam development was because of involvement I had in that exam which did require me a wait 2 years or so before taking the exam which I have more than met. I know others who had taken and passed that exam and it's a fairly rigorous test for a building designer certification and we wanted it to be valid, defensible, and because we were working on ANS accreditation for the exam which is in the process. Takes awhile to do. Ambitious for a small organization like the AIBD but we have been working hard with significant money put into the work of ANS accreditation. 

    Exam development processes are somewhat standardized so what we went through, it would be similar to what NCARB goes through so I understand in general what goes on in the process. Yes, there be differences in the details but the basic process is substantively similar. 

    REGARDING NCARB and ARE 5.0/5.1

    I see some of the issues people have been mentioning even in the online demo exam for the current ARE 5.0 currently tested. Since it will be the same "Exam Studio" (okay.... an updated version of it) software as a service (SaaS) platform. Any bugs, glitches, or otherwise that rests in "Exam Studio" itself in the ARE may still be present unless Zoomorphix addressed them in the process. Zoomorphix is a software company so they should be able to address them. They are the people to address those issues. NCARB is responsible to make sure Zoomorphix is aware of those bugs, glitches, or whatever, and come up with a solution to fix or address them. This can be done. There are things that can be done to alleviate slow loading PDF but if the test centers' computers are loaded with heavy impacting real-time firewalls, real-time virus/malware scanners, and other such things as those, there is not much anyone can do to get such things running fast because of the computer security software. They are normally expected to slow things down. Either those are turned off/uninstalled or they find a different security software solution that doesn't bog down internet traffic speed. 

    A little about myself, I have a background in software development since the 1980s. 

    In my opinion, it is possible to fix bugs and such that have been mentioned if the software developers are able to reproduce those bugs/glitches on their end and then fix them. That can be done by a small team of 3-5 software people in a month's time. Adding something like digital whiteboard is going to take considerably longer and possibly a project team of about 5-10 individuals. A 2-3 programmers, 1-2 graphic artist(s) (mostly technical graphic artists to design the UI elements), in-house tester or two, QA (Quality Assurance) individual(s), and project manager/director-"software architect/engineer" (Project lead) and assistant project manager. The bug fixers are likely just going to need an additional 1-2 programmers and overseen by Project Lead and his/her assistant project manager. So a team of about 12 should be sufficient to fix issues like that seen if they were able to reproduce the results and are able to find the problem is rooted in their software.

    NCARB is responsible for the exam items like what the test items are as prepared by item writers and the supporting content that is explicitly part of the ARE test even though the software maybe used for other exams not related to the ARE. 

    I'm not going to be biting off the heads of Zoomorphix or make judgment on their work on the updates until I see it. I can criticize them for their existing work on the ARE 5.0 but it serves no one. Flaws are more systemic of the disjointed process and communication problems which I think we all see there is. 

    I think we ALL agree that it is in the benefit of EVERYONE (including NCARB) that the software works and the exam experience is good (not easy.... rigorous... but good in that the exam software and computer doesn't f--- the examinee and cost them time and money). We aren't trying to make an exam that every test taker will pass with ease. The prepared should pass these exams but those who aren't ready are the ones that should not easily pass but it should be their lack of knowledge and skills NOT some buggy, glitchy software glitching or crashing multiple times. If I fail the exam, it should be me not the software that cause the fail. It should be as reliable as the old fashion paper and pen/pencil exams. Those don't "crash".

     

     

    1
    Comment actions Permalink
  • Avatar
    Richard Balkins

    NCARB,

    As Elif Bayram pointed out, there is a lack of confidence in the parties involved:

    Zoomorphix, the developer of the Exam Studio SaaS that delivers the ARE 5.0 exam. This exam is buggy, glitchy, and crashes on the test center's computers. Do these developers actually test the software out at test center facilities on their computers? It concerns the examinees because these are the people who made the buggy and glitchy exam so they don't trust them to implement this "white board" feature without it glitching and crashing.

    They better test the software at ALL the test centers and check that they work flawlessly. If they don't, FIX them and RETEST and not deployed for actual examinees until it is working flawlessly. Then they also get 150 architects to volunteer their time to test the exam software. It is not about the answering the questions correctly but making sure the features work. If they work flawlessly in a remote proctoring setting but are glitching at the test centers then we need to assess what's different. Is it the age of the computers? Is it an outdated version of Windows? Is it the security software conflicting with the exam software. It is already know that firewalls can be an issue but there is no test center with hundreds of computers on a local area network that would ever be setup without firewalls. There is even firewalls in the managed switches, routers, and services gateway used at these data centers and the actual server. You need to test the software in that kind of network environment setting.

    They might work flawlessly in Zoomorphix's facility but they are not necessarily setup with a most test center lab matching Prometric test centers. Hell, one test center may not have the exact same computers at a test center 1000 miles away. They may have different monitors. They may have different computers & motherboards. They may have different ISP providers. It's all kind of ad hoc and they need to go to this level of testing to certify that their software works in these testing environments.

    It's clear that there is low confidence of Zoomorphix because its the same folks that developed the buggy & glitchy exam software that crashes and to hear (read) that they are the same folks implementing the digital whiteboard. The same folks that made the highlight or strike tool that crashes the exam software are the same folks to write a digital whiteboard. Really, if they can't make a highlight or strike tool that works without crashing the exam, how are they going to be able to make a digital whiteboard that doesn't crash the exam? That's the message. 

    We want proof not words but actions and proof. Zoomorphix needs to prove themselves to us before we trust them. As a software developer, I would flag that as a quality assurance failure. ARE 5.0 should never have been released until the quality assurance had been done and implemented. In fact, Zoomorphix should have done this before going to NCARB and others marketing their Exam Studio software. They should have made sure the software is reliable to mission critical like NASA standards. "Failure is not an option" as the saying often attributed to Gene Kranz (famous flight director of NASA). I wish NCARB and Zoomorphix applies a little bit of Gene Kranz principles to the exam software development as if life depends on it. 

     

    7
    Comment actions Permalink
  • Avatar
    Richard Balkins

    No worries.
    Looking through your reasoning why the AIBD exam is open book, it looks as though you have made a beautiful case as to why the AREs should be open book, but we know they won't do that. It's too fair.
    Appreciate your researching the software too. Do you have any idea why exams keep crashing and if there's any way NCARB could fix the problem? It's been doing it for over a year and all NCARB will say on it when we ask is "we need to do better" maybe you could provide some perspective as to why they won't fix it (other than because it would cost money and they are in the business of making money and not spending it)
    Once you start the AREs I'm sure you'll do well.

    I wasn't really focusing on suggesting ARE to be open book but it is an option. I'm kind of guessing a little because there can be so many possible causes and that there are multiple issues that people mentioned but I don't have a list which would help actually but to a point of course. I don't have the source code exactly of exam studio and there is possible system differences between deploying the exam through the test center and the demo exam which is a demo exam after all. I noticed some things. I don't want to get too close to the "exam software" to compromise my authorization to test or cause a delay or lockout from exam taking for like 2-3 years.

    NCARB, itself, probably can't fix software at all. Lacking of software programmers. However, Zoomorphix SHOULD be able to fix it. They have the source codes for any software behind the scenes on servers. They wrote the Exam Studio SaaS software. If I say the code itself, I could possibly find the problems for any that lies within the software. However, if the proctor browser (eg. ProProctor) you use to connect to the ARE exam crashes then we have to direct that to Prometric and their software team/consultants. Actually, we direct it to NCARB and they are suppose to direct it to the respective parties. If I see the issues, I'll direct them to the parties. 

    Can there be a conflict between the proctor's browser and the Exam Studio SaaS (ARE 5.x)? Yes. For example, if the browser crashes while processing some of the client-side scripts then yes that can be a problem but we have to direct the issues to the right parties and each party involved implement them. 

    Can they be fixed? Yes. Anything software-wise can be fixed. In software development, when receiving bug reports, the quality of information including exact steps to reproduce the bug (like the crashes) is crucial for effective bug-fixing. Merely reporting a bug or crash issue can take awhile to reproduce the error because every software 'glitch' or 'bug'/crash is caused be specific reasons and if they 'glitch', they should be able to be reproduced. Once reproduced, we can find a fix because we can understand the underlying problem and find a solution.

    If NCARB is not relaying the information accurately or with the necessary information to reproduce the bugs or glitches. This may mean we have to reproduce the testing center's 'environment' from a computer & networking systems environment. 

    Sometimes the software uses underlying APIs of the computer and Microsoft is notorious for mucking with the operating systems and APIs through their Windows updates so sometimes a dependency that a particular feature in the exam software (Proproctor or Exam Studio) relies on (dependencies) are changed or replaced because it is deprecated, it can cause all kinds of trouble. 

    It isn't easy but it is fixable stuff.

    If we can reproduce some or all those glitches/bugs/crashes in the demo exam and it is video recorded, it might help. 

     

    0
    Comment actions Permalink
  • Avatar
    Lilia Grigoryan (Edited )

    Richard,

    I appreciate your long and detailed posts. 

    Let’s start from the demo exam on this website....The highlighting and strikeout tools are not working on the demo exam.

    I would like to point out that the candidate has a direct contract with NCARB and not with all other companies that you were mentioning above. NCARB and ARE candidates must be in one team, not against each other.

    By summarizing your posts, NCARB lacks accountability and transparency.

    2
    Comment actions Permalink
  • Avatar
    Richard Balkins

    For an argument for support of an open-book or equivalent: https://ncees.org/wp-content/uploads/ExamineeGuide_June-2020.pdf

    ARE does a little.

    NCEES CBT has this:

    The current version of the appropriate NCEES
    reference handbook will be supplied onscreen
    as a searchable PDF. All NCEES examinees
    will use a 24-inch monitor while testing to
    allow sufficient space to display both the exam
    question and the reference handbook. You can
    access the current version of the handbook from
    your MyNCEES account.

    In other words, they supply in reference material need and so forth. 

    In the case of the CPBD, while it is open-book, you have 175 questions (150 of them scored items so lets say 150 questions) that you have to answer in 3 hours. You don't have the time to use the reference books to look up your answers. You do need to know the subject matter well enough to not depend on it for every question. 

    If the ARE is made open-book or have comprehensive reference material that you can use to answer the question, it should be fair. The ARE should be fair yet rigorous and I don't care if the person takes the exam at a test center, at home, or at their office (as long as it is suitable and meets some standards for exam integrity). 

    An open-book exam isn't necessarily less rigorous or that it would be basis for cheating. Cheating issues can be addressed adequately for open-book exams regardless of whether the test taker is proctoring at home or at a test center. There are already good policies for that stuff. An ARE test taker  in an (open-book ARE) would not be able to simply look into the books and find an answer. Case Study model works well for this with clear right answer(s) and others wrong. Proper item types and the process of thinking a solution. 

    In the NCEES case for CBT testing, they provide a "reusable booklet" and marker. Providing something like that for ARE at test centers would be appropriate while allowing scratch papers following a protocol similar to CPBD with a requirement that those papers are shredded during checkout with cross-cut paper shredder because we can have an inspection procedure to examine shreds because your most cheapest paper shredders makes long strips while a cross-cut paper shredder cuts the paper into smaller pieces because it cuts the paper perpendicular to the strip cut so it's cut into tinier squares/rectangular pieces and some are really small and way too much work to tape back together. This would make things relatively equitable for all test takers AND allow time for the digital whiteboard feature to 'mature' and get better.

    A side of me wants NCARB to succeed in remote proctoring but we need to be fair for all and the exam items needs to be such that we aren't doing overly crazy stuff. I can support open-book with the right check-in procedures. I think you may have skimmed through the CPBD candidate handbook for that exam and appropriate protocols can be employed. The questions are already written in such a way that you can't really simply look at a book and get the answers. It can provide you information like equations to use to solve a mathematical based test item but you have to do the work and determine the right answer from the list of answers.

    It can still be valid, defensible, and meet the rigorous needs of the ARE. The ARE doesn't need to be "open-book" if it provides ALL the information a test taker needs to answer the questions without necessarily outright giving them the answer. Adequate time is needed but not too much and not too little. 

    Therefore, it should be fair and I think most of us wants that. The big issue is the argument is what is "fair". 

     

    0
    Comment actions Permalink
  • Avatar
    Richard Balkins

    Richard,

    I appreciate your long and detailed posts. 

    Let’s start from the demo exam on this website....The highlighting and strikeout tools are not working on the demo exam.

    I would like to point out that the candidate has a direct contract with NCARB and not with all other companies that you were mentioning above. NCARB and ARE candidates must be in one team, not against each other.

    By summarizing your posts, NCARB lacks accountability and transparency.

     

    Lilia, I agree with you. I don't like to pester the software vendor because we are not party to contract with them. Our contract is with NCARB but if NCARB is ineffective then its the next step to get things fixed. 

    I think for most of us, it isn't really at all about the making available remote proctoring. Most of us wouldn't have a problem with it if it doesn't cause those taking exam at test center a bunch of trouble. Accountability, transparency, and communication is a problem. 

    People were expecting the remote proctoring experience to be pretty much the same as test-center with scratch paper with modifications to the check-in and check-out procedures which would be expected. They had been provided information about the CPBD candidate handbook since August 18, 2020. It only takes a few minutes to read the pertinent pages.

    I want a commitment on NCARB to take efforts to fix the bugs, glitches, crashes that may still exist in the new exam and new ones and do a better job at relaying those issues to the pertinent parties. I mean relay of information in 2-3 business days which is fair and reasonable because the SaaS provider should be able to patch or fix the exam software with scheduled update maintenance cycles which should be semi-regular and notices sent in advance so examinees know about it with plan down-time hours which should be done fairly and with least impact. This will allow the exam to get updated. For example, exams being Monday through Sat. but Sunday being an update cycle. This would help keep the software updated. Additionally, the exam browser app needs to be kept updated and fixes done from time to time so they run reliable.

    Test centers are responsible for their computers just like the rest of us are responsible for our own computers like running OS updates, etc. Those should be done a day or so before any exam. Our browser app client should notify us when there is updates and to update it just like Chrome/Chromium browser does, Steam client does, any MMO game client does, etc. 

     

    1
    Comment actions Permalink
  • Avatar
    William Dubois

    I appreciate NCARB working to provide more options so people can test more safely or conveniently if they are in remote locations. However I have several frustrations and concerns about this switch.

    In the NCARB statement we are told, "Disparate treatment of candidates is unethical and indefensible." Yet I am left weighing the advantage of using pencil and paper for my upcoming exam or if I should delay it until after the switch so I can take an exam with less questions and more time to answer. The difference in amount of time or questions has not been clarified. Are we talking about 5 minutes? 50 minutes?

    My second concern is the software. During my last exam the program froze up and I had to switch to a second computer. The on-staff Prometric proctor was able to set up the second computer and resume the exam, but will candidates be expected to run tech support or have a second computer ready? Will there be live support during all testing in case an issue like this occurs?

    The presentation of the case study information is already obtuse and unwieldly. I don't know exactly how the virtual whiteboard operates but imagining using the white board in combination with the case study format gives me headaches. Is there an example from a similar product of how this might work or how it will look already out there? My next test is in mid October so if I wait to see how this will function it might be after my next exam.

    Which leads me to another question. Will online test takers be able to use a second screen for their online tools? The higher quality screen and mouse I have at home/ work (the scroll wheel was buggy on the last testing computer) would already be a nice advantage over the testing center. 

    I am trying to weigh the advantages and disadvantages of the new format so I appreciate the responses.

    1
    Comment actions Permalink
  • Avatar
    Kari Scuglik

    I would like to echo this:

    WE the test takers would like a digital format that matches that of pen and paper. If it means we have to strap a Wacom (or similar mouse trackpad like) tablet in so we can easily control the mouse on the screen like its the tip of a pencil, so be it. I think that would be the best case scenario to make literally everyone happy. WE the TEST TAKERS would like to be able to see this whiteboard on the screen 24/7. It cannot vanish, cannot be automatically erased, cannot glitch, and cannot cause us stress. Paper does not cause us stress. We will need tools within this whiteboard that accurately allow us to do complex structural calculations and draw crude diagrams within seconds. Every second counts on these exams and it's important that this whiteboard works with us, not against us.

    Also, the calculator tool IS UNRELIABLE AND UNRESPONSIVE and the slow-loading case study PDFs are so bad - any "white board" based on the same technology will most likely have the same frustrating problems.

    2
    Comment actions Permalink
  • Avatar
    Richard Balkins

    Your computer at home (if it is newer with a faster (more cores and/or GHz speed) with more ram, a real video card, and a dedicated 100 Mbps or faster broadband internet service to your home, simpler networking infrastructure, and less demanding internet security software (firewall, realtime antivirus/antimalware scanners, etc.).... likely your computer can be a lot more reliable. Prometric and any other proctor is responsible for their computers and you would be with remote proctoring. If Prometric is using outdated computers that are decade old or something then that's a legitimate complaint that NCARB should be on their proctors about updating. There are some issues with the exam software which will need to be addressed.

    I won't be having the usual Norton and McAfee garbage or other crud like DeepFreeze that you might find on computers institutionally setup in a large network environment (50+ computers). Therefore, I would have a more 'clean' connection with less things to get in the way.

    In a large network, you are usually all feeding traffic into a server. It's very similar to what you would expect in a college computer lab environment. All I would have would be the direct streamline connection between computer to router(s) to cable-modem. No Internet Server beastie that runs antivirus/antimalware scan on every single TCP/IP or UDP packet going in and out.

    The exam may actually work better or more reliably in remote setting until test centers do stuff about their computers and network. The exam software developer will have to address the software bugs/issues that is part of the exam software itself. Some people have mentioned some of those issues. This may actually help diagnose issues and where the issue resides. Software bugs/glitches like flaws in the calculator will appear in remote proctoring (or the theory is) which we will know is the exam software. It will isolate test center IT flaws from the exam software flaws. (That's the theory)

    I hate being the guinea pig on my dime. 

     

    -1
    Comment actions Permalink
  • Avatar
    Sean Dorsy (Edited )

    NCARB needs to spend less time hindering the growth of all candidates because a few might cheat, and instead, actively promote growth so no candidate feels the need to cheat.

    Provide us with relevant testing information and resources. Provide us with functioning testing software. Provide us with detailed score feedback. Provide us with the tools to succeed, rather than unnecessary hurdles. It is absurd how little they prepare a candidate for their exam--instead, they refer us to thousands of dollars of textbooks with little direct coordination to the practice of architecture. NCARB is rewarded when we fail, and in turn, so is LineUp. As mentioned by Jennifer, NCARB employees worked on the creation of LineUp while being employed by NCARB. Meaning, candidates paid for the creation of private software which will be packaged up and sold back to us.

    The passing rate for ARE 5.0 is ~50% (when the 6 exams are averaged together). FIFTY PERCENT. That is abysmal--yet they still collect our annual membership and testing fees. For comparison, the Bar exam and Medical Licensing Exam have a passing rate of ~90% for first-time test takers. The ARE's ~50% passing rate is for all tests taken, meaning, even after multiple tries candidates are still failing. 

    Let's be clear--we are not failing NCARB's exams, NCARB is failing us as candidates by not measuring up to their own tag line ("Helping licensure candidates... reach their career goals."), and continue to fail at accomplishing their own mission statement and core values. NCARB is a hindrance to candidate career development and will continue to negatively impact the practice of architecture unless we take a stand. Yes, examinations are needed to prove a candidate's competency to protect the health, safety, and welfare of the public. That said, competency is also required within the accrediting boards and testing body to effectively administer the exams.

    NCARB: Repay all candidate membership dues since the implementation of ARE 5.0 and repay all fees collected from failed 5.0 exams. You can not claim to be helping us reach our career goals when you actively prevent us from achieving our goals through your continual mediocrity. 

    3
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk